Domain-specific method for distinguishing type-denoting domain terms from entity-denoting domain terms

ABSTRACT

Large lists of domain-specific terms are classified as a particular kind of linguistic object, e.g., lexical answer type T versus canonical answer E, based on features from a domain-specific corpus which have been found to distinguish between the linguistic objects. The distinguishing features can be identified in the corpus based on sets of the linguistic objects derived from question-and-answer pairs. A classifier can be trained using the distinguishing features, and the classification carried out using that classifier. The distinguishing features can include one or more syntactic features or one or more lexical features. The linguistic objects (the T and E training sets) can be extracted from the question-and-answer pairs automatically via text analysis if manually curated lists are not available. The classified terms can be included in a domain-specific lexicon which facilitates a deep question answering system to yield an answer to a question.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with United States Government support underagreement no. 2013-12101100008. THE GOVERNMENT HAS CERTAIN RIGHTS INTHIS INVENTION.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention generally relates to natural language processing,and more particularly to a method of analyzing text to categorize largesets of domain-specific terms.

Description of the Related Art

As interactions between humans and computer systems become more complex,it becomes increasingly important to provide a more intuitive interfacefor a user to issue commands and queries to a computer system. As partof this effort, many systems employ some form of natural languageprocessing. Natural language processing (NLP) is a field of computerscience, artificial intelligence, and linguistics concerned with theinteractions between computers and human (natural) languages. Manychallenges in NLP involve natural language understanding, that is,enabling computers to derive meaning from human or natural languageinput, and others involve natural language generation allowing computersto respond in a manner familiar to a user. For example, a non-technicalperson may input a natural language question to a computer system, andthe system intelligence can provide a natural language answer which theuser can hopefully understand. Examples of an advanced computer systemsthat use natural language processing include virtual assistants,Internet search engines, and deep question answering systems such as theWatson™ cognitive technology marketed by International Business MachinesCorp.

Text analysis is known in the art pertaining to NLP and typically uses atext annotator program to search text documents (corpora) and analyzethem relative to a defined set of tags. Text annotators and corpora canbe domain-specific, that is, intended for use in a particular context ofinterest such as medicine, business processes, sports, etc. The textannotator can generate linguistic annotations within the document to tagconcepts and entities that might be buried in the text. A cognitivesystem can then use a set of linguistic, statistical andmachine-learning techniques to analyze the annotated text, and extractkey information such as person, location, organization, and particularobjects (e.g., vehicles), or identify positive and negative sentiment.Front-end NLP can include identification of a lexical answer type and afocus among others. A lexical answer type (LAT) is a term in a questionthat indicates what type of entity is being asked for, i.e., the primaryconcept that is being discussed. Focus is essentially the subject of thetext or, in the case of a question, the answer to the question or areference to the answer (an entity). For example, a LAT in a questionmight be a person type, with the answer being a specific person.

SUMMARY OF THE INVENTION

The present invention in at least one embodiment is generally directedto a method of distinguishing at least two classes of domain-specificterms that are crucial to the domain-specific natural languageprocessing involved in deep question answering—a set T ofdomain-specific terms that refer to domain entity types and a set E ofdomain-specific terms that refer to domain entities. This isaccomplished by making use of a training set T′ of domain terms known torefer to domain entity types and a set E′ of domain terms known to referto domain entities to identify distinguishing features from one or morecorpora specific to a particular domain wherein the distinguishingfeatures distinguish the linguistic objects in T′ from the linguisticobjects in E′, and using these features to classify the terms from alist specific to the particular domain. In the illustrativeimplementations an automatic machine-learning classifier can be trainedusing the distinguishing features, and the classifier can then be usedto classify the terms from the domain specific terminology list. Thedistinguishing features can include one or more syntactic features orone or more lexical features. The training sets (the P and E′ sets) canbe extracted from the question-and-answer pairs automatically via textanalysis if manually curated lists are not available. The classifiedterms can be included in a domain-specific lexicon which facilitates adeep question answering system to yield an answer to a question.

The above as well as additional objectives, features, and advantages inthe various embodiments of the present invention will become apparent inthe following detailed written description.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be better understood, and its numerousobjects, features, and advantages of its various embodiments madeapparent to those skilled in the art by referencing the accompanyingdrawings.

FIG. 1 is a block diagram of a computer system programmed to carry outnatural language processing, including domain-specific termclassification, in accordance with one implementation of the presentinvention;

FIG. 2 is a table of domain-specific question-and-answer pairs fromwhich linguistic objects are extracted in accordance with oneimplementation of the present invention;

FIGS. 3A and 3B are tables of lexical answer types (7) and answerentities (E) extracted from the question-and-answer pairs of FIG. 2 inaccordance with one implementation of the present invention;

FIG. 4 is a block diagram of a classifying system constructed inaccordance with one implementation of the present invention whereinlinguistic objects extracted from the question-and-answer pairs are usedto identify distinguishing features from a domain-specific corpus, andthose distinguishing features are then used to train a natural languageclassifier;

FIG. 5 is a block diagram showing the use of the natural languageclassifier to categorize large sets of terms which can then be used tosupport a deep question answering system in accordance with oneimplementation of the present invention; and

FIG. 6 is a chart illustrating the logical flow of a classificationprocedure in accordance with one implementation of the presentinvention.

The use of the same reference symbols in different drawings indicatessimilar or identical items.

DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

Deep question answering systems make a distinction between terms thatrefer to types of entities and terms that refer to entities. These twoclasses of terms play important roles in the processing mechanisms builtinto deep question answering systems, and provisioning a deep questionunderstanding system with adequate domain-specific lexical resourcesthat articulate this distinction for a specific domain is one of thecrucial ways in which domain adaption of such systems proceeds. Termsthat refer to types are often good candidates for the lexical answertype (LAT) of a question, while terms that refer to entities are oftengood candidates for the answer itself. This distinction can be crucialto answer generation, answer scoring, answer filtering and othercomponents of deep question answering. While a given term might bothrefer to a type of entity and refer to an entity, for a particulardomain, terms that make good answer types tend to make bad answers andterms that make good answers are generally bad types.

In adapting a deep question answering system to a given domain, subjectmatter experts often provide lists of words and multi-word terms thatare relevant to their domain, and these terms must be sorted into termsreferring to types and terms referring to entities for them to be usedappropriately in the deep question answering lexicon. Experience hasshown that subject matter experts have difficulty making thisdistinction, and that it is a time consuming task for domain adaptationlanguage technology experts. This distinction is often highlydomain-specific. For example, the word “protein” may have a differentrole to play for question answering in the cancer research domain thanin the body building domain. These different roles can be seen bycomparing some question-and-answer (QA) sets for such domains. Here aretwo sample body building domain QAs where “protein” is an answer.

-   -   Question: What can I add to my diet to build muscle?    -   Answer: Protein is the cornerstone of my bodybuilding nutrition        plan in that it determines how many meals I eat each day.    -   Question: What is seafood is an excellent source of?    -   Answer: Seafood is an excellent source of protein and it's        usually low in fat.

Here are two sample cancer research domain QAs where “protein” is a LAT.

-   -   Question: What kinds of proteins act as immune system targets?    -   Answer: Researchers have spotted rare ‘flag’ proteins that act        as immune system targets and are displayed on the surface of all        of a patient's tumor cells, wherever they might be in the body.    -   Question: What two proteins did a Stanford team use to stop        metastasis, without side effects?    -   Answer: The Stanford team seeks to stop metastasis, without side        effects, by preventing two proteins—Axl and Gas6—from        interacting to initiate the spread of cancer.

In customizing a deep question answering system with thousands of termsto be added, determining what role a term will play is thus a criticaltask, which feeds not only the deep question answering system itself,but also provides useful feedback to the domain adaptation team as topotential gaps in the taxonomy that should be filled, for example, typenames that have only a few answer-level (entity) names associated withthem. However, customer-provided lists of domain-specific terms are verytime-consuming to sort into categories that are required for NLPsystems. It would, therefore, be desirable to devise a method ofautomatically categorizing large sets of domain-specific terms. It wouldbe further advantageous if the method could leverage other resourcesalready available as part of front-end NLP.

The present invention achieves these objectives by leveraging existingartifacts involved in the domain adaptation task to automaticallyclassify domain terms into those that refer to entities and those thatrefer to entity types. In exemplary implementations, this would involveextracting training sets of linguistic objects from domain-specificquestion-and-answer pairs, identifying features from a domain-specificcorpus which can be used to distinguish these sets of linguisticobjects, and using these features to classify domain terms in a largelist of terms as being one of the particular linguistic objects, e.g.,either a “likely LAT” or a “likely entity”.

With reference now to the figures, and in particular with reference toFIG. 1, there is depicted one embodiment 10 of a computer system inwhich the present invention may be implemented to carry out naturallanguage processing including domain-specific term classification.Computer system 10 is a symmetric multiprocessor (SMP) system having aplurality of processors 12 a, 12 b connected to a system bus 14. Systembus 14 is further connected to and communicates with a combined memorycontroller/host bridge (MC/HB) 16 which provides an interface to systemmemory 18. System memory 18 may be a local memory device oralternatively may include a plurality of distributed memory devices,preferably dynamic random-access memory (DRAM). There may be additionalstructures in the memory hierarchy which are not depicted, such ason-board (L1) and second-level (L2) or third-level (L3) caches. Systemmemory 18 has loaded therein various NLP tools, including termclassifier tools as taught herein.

MC/HB 16 also has an interface to peripheral component interconnect(PCI) Express links 20 a, 20 b, 20 c. Each PCI Express (PCIe) link 20 a,20 b is connected to a respective PCIe adaptor 22 a, 22 b, and each PCIeadaptor 22 a, 22 b is connected to a respective input/output (I/O)device 24 a, 24 b. MC/HB 16 may additionally have an interface to an I/Obus 26 which is connected to a switch (I/O fabric) 28. Switch 28provides a fan-out for the I/O bus to a plurality of PCI links 20 d, 20e, 20 f These PCI links are connected to more PCIe adaptors 22 c, 22 d,22 e which in turn support more I/O devices 24 c, 24 d, 24 e. The I/Odevices may include, without limitation, a keyboard, a graphicalpointing device (mouse), a microphone, a display device, speakers, apermanent storage device (hard disk drive) or an array of such storagedevices, an optical disk drive which receives an optical disk 25 (oneexample of a computer readable storage medium) such as a CD or DVD, anda network card. Each PCIe adaptor provides an interface between the PCIlink and the respective I/O device. MC/HB 16 provides a low latency paththrough which processors 12 a, 12 b may access PCI devices mappedanywhere within bus memory or I/O address spaces. MC/HB 16 furtherprovides a high bandwidth path to allow the PCI devices to access memory18. Switch 28 may provide peer-to-peer communications between differentendpoints and this data traffic does not need to be forwarded to MC/HB16 if it does not involve cache-coherent memory transfers. Switch 28 isshown as a separate logical component but it could be integrated intoMC/HB 16.

In this embodiment, PCI link 20 c connects MC/HB 16 to a serviceprocessor interface 30 to allow communications between I/O device 24 aand a service processor 32. Service processor 32 is connected toprocessors 12 a, 12 b via a JTAG interface 34, and uses an attentionline 36 which interrupts the operation of processors 12 a, 12 b. Serviceprocessor 32 may have its own local memory 38, and is connected toread-only memory (ROM) 40 which stores various program instructions forsystem startup. Service processor 32 may also have access to a hardwareoperator panel 42 to provide system status and diagnostic information.

In alternative embodiments computer system 10 may include modificationsof these hardware components or their interconnections, or additionalcomponents, so the depicted example should not be construed as implyingany architectural limitations with respect to the present invention. Theinvention may further be implemented in an equivalent cloud computingnetwork.

When computer system 10 is initially powered up, service processor 32uses JTAG interface 34 to interrogate the system (host) processors 12 a,12 b and MC/HB 16. After completing the interrogation, service processor32 acquires an inventory and topology for computer system 10. Serviceprocessor 32 then executes various tests such as built-in-self-tests(BISTs), basic assurance tests (BATs), and memory tests on thecomponents of computer system 10. Any error information for failuresdetected during the testing is reported by service processor 32 tooperator panel 42. If a valid configuration of system resources is stillpossible after taking out any components found to be faulty during thetesting then computer system 10 is allowed to proceed. Executable codeis loaded into memory 18 and service processor 32 releases hostprocessors 12 a, 12 b for execution of the program code, e.g., anoperating system (OS) which is used to launch applications and inparticular the NLP application of the present invention, results ofwhich may be stored in a hard disk drive of the system (an I/O device24). While host processors 12 a, 12 b are executing program code,service processor 32 may enter a mode of monitoring and reporting anyoperating parameters or errors, such as the cooling fan speed andoperation, thermal sensors, power supply regulators, and recoverable andnon-recoverable errors reported by any of processors 12 a, 12 b, memory18, and MC/HB 16. Service processor 32 may take further action based onthe type of errors or defined thresholds.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Computer system 10 carries out program instructions for natural languageprocessing that uses novel analysis techniques to manage theclassification of large lists of domain-specific terms. Accordingly, aprogram embodying the invention may include conventional aspects ofvarious NLP tools, and these details will become apparent to thoseskilled in the art upon reference to this disclosure.

In many deep question answering systems, the system is tuned to specificapplication domains by engaging in a process known as domain adaptation.This task is usually performed by an experienced NLP analyst working inconcert with an expert in the particular domain of interest. In atypical domain adaptation exercise, domain experts are called upon tosubmit long lists of domain terms for ingestion into the system. The NLPanalyst then assesses the list to create domain-specific dictionaries,on the basis of general knowledge about the role of domain dictionariesin the system and additional knowledge of the domain, distinguishingterms that may be answers from terms that refer to types of answers.This task is difficult and time consuming, and adequately tuning thedomain dictionaries is a significant problem that calls out for ansystematic solution. The present invention addresses this problem.

In addition to generating terminology lists, domain experts often createreasonably large sets of question-and-answer pairs that reflect thekinds of domain-specific questions that users of the deep questionanswering system might be expected to put to their system as well asidentifying domain-specific document sets (corpora) that will containanswers to these questions. The current invention can leverage these QApairs and these corpora to classify the domain terms as summarizedabove. This classification method is specific to a given domain andcorpus. The general idea is to develop a text classifier to distinguishelements in the domain terms list into at least two linguistic classes,particularly a type (T) class and an entity (E) class. Training data forthis classifier is derived from the QA pairs, with the identifiedlexical answer types from the questions serving as T-class ground truthand the identified answer entities to the questions serving as E-classground truth. In this manner, domain-specific training data can beapplied to domain-specific corpora to derive a domain-specificclassifier that can distinguish domain terms into domain T terms (thoseterms that are used in that domain typically as types of answers) anddomain E terms (those terms that are used in that domain typically asanswers to questions).

Referring now to FIG. 2, there is depicted an exemplary set 50 ofdomain-specific question-and-answer pairs. For this example, the domainis world geography. The QA pairs can be curated by any means, includingmanual, or using collections of previously derived QA pairs. There arepreferably hundreds of QA pairs in set 50. The QA pairs may include aprevious identification of LAT terms and answer entities, or they can beexamined by computer system 10 using conventional text analysis toautomatically identify these and other types of linguistic objects. Forexample, named entity recognition is known in the art and useslinguistic grammar-based techniques as well as statistical models, i.e.machine learning, to annotate sentences (including questions). The QApairs can be stored on computer system 10 or remotely.

Terms can be extracted from the multiple QA pairs by computer system 10and assigned into one of at least two sets (T and E) as further seen inthe tables 60, 62 of FIGS. 3A and 3B. FIG. 3A shows the set T of LATsextracted from the QA pairs, and FIG. 3B shows the set E of entitiesextracted from the QA pairs. For example, the first QA pair in table 50are “What country has the most people?” and “China has the world'slargest population.” From these sentences, the term “country” has beenidentified as a LAT and added to table 60, while the word “China” hasbeen identified as an entity and added to table 62. Other extracted LATsinclude “mountain”, “rainforest”, “ocean”, “lake”, and “river”, andother extracted entities include “Mount Everest”, “Amazon River Basin”,“Marianas Trench”, “caldera”, and “Amazon”. As with table 50, there canbe hundreds or even thousands of entries in tables 60, 62. The T and Esets can also be stored on computer system 10 or remotely.

FIG. 4 shows how the T and E tables 60, 62 can be used in oneimplementation of the present invention to identify features of thedomain of interest which can in turn be used to distinguish terms asdifferent linguistic objects. A feature identification module 72 runningon computer system 10 takes the terms from the T and E tables 60, 62 andsearches for those terms with a domain-specific corpus or corpora 74.Computer system 10 can then examine the usage of the particular terms asfound within corpora 74 to identify features 76 which appear to becommon to one class or another (LAT or entity). Any feature havingstatistical significance can be used, particularly syntactic featuresand lexical features. For example, a syntactic feature might be ‘appearsas the subject of a sentence’ (e.g., “Protein is good for you’) or‘appears as the possessor phrase’ (e.g., “Lincoln's wife was strange.”).Syntactic-lexical binary features can also be used, e.g., the termoccurring before the phrase “such as” or occurring after the phrase“kinds of”, ngrams (a contiguous sequence of items from a given snippetof text), or combinations of any of the foregoing. These distinguishingfeatures can be used to build a type-entity classifier 78 which istrained on the two sets T and E. Classifier 78 can also be stored oncomputer system 10 or remotely.

FIG. 5 illustrates how the type-entity classifier 78 thus constructedcan be further used to generate a domain-specific lexicon or dictionary82 in accordance with one embodiment of the present invention.Classifier 78, running on computer system 10, receives a list of terms84 pertaining to the domain of interest, and uses the distinguishingfeatures (also domain-specific) to classify each term in list 84 aseither a “likely LAT” or a “likely entity”. In one embodiment, forexample, the classifier can be based on features reflecting commonsyntactic contexts of a term as it appears in the corpus (wheresyntactic context might be distinguished by the sequence of words beforeand after the term, and the frequency of the context might be a count ofthe number of times the same words appear before and after words in adesignated class). Using these kind of features, the N most frequentcontexts in which terms on the T-class ground truth list appear would beextracted from the corpus along with the N most frequent contexts inwhich terms in E-class ground truth list appear. A target term from thedomain terms list might be classified by determining if its distributionwithin a domain-specific corpus (such as in corpora 74) is more like theT-class terms or the E-class terms (in the simplest case by counting howmany of the T-class frequent contexts it appears in and how many of theE-class frequent contexts it appears in). Other potentialcorpus-specific classification methods could be used. The resultinglexicon 82 includes an appropriate tag for each term indicating itsdetermined class, and can then be used by a deep question answeringsystem 86 to facilitate the provision of a natural language answer to anatural language question. Deep question answering system 86 can also berunning on computer system 10.

One example of a way in which these tags can facilitate the deepquestion answering system is in answer scoring. In many deepquestion-answering systems—such as Watson™ systems—one component of theprocess involves determining whether a term identified as a possibleanswer to the question is of the right type. So in the case of “Whichsubstance was used by Stanford to . . . ?”, much of the processinginvolves identifying candidate answers (such as “Gas6”); if we know thatin the given domain there is a type “protein”—which is a substance—andthat “Gas6” is an entity of this type, then that answer would be highlyscored and returned as a good result.

The present invention may be further understood with reference to thechart of FIG. 6 which illustrates the logical flow for a classificationprocess 90 in accordance with one implementation of the presentinvention, which may be carried out on computer system 10. Process 90begins by extracting sets of linguistic objects from question-and-answerpairs (92). There must be at least two kinds of linguistic objectsextracted, such as lexical answer type and answer entity. Features froma domain-specific corpus are identified which distinguish the kinds oflinguistic objects so extracted (94). These distinguishing features canbe based on various statistical measures of different usages of theobjects, particularly syntactic or lexical contexts. Terms in largelists can then be automatically classified, e.g., as either LAT oranswer based on the distinguishing features (96). In the illustrativeembodiment, this step is carried out with a classifier trained with thedistinguishing features.

The present invention thereby provides an efficient and effective methodof categorizing very large sets of terms associated with a particulardomain. This approach not only saves countless hours of manualclassification, but further provides a more robust lexicon which canhelp a deep question answering system provide superior results.

Although the invention has been described with reference to specificembodiments, this description is not meant to be construed in a limitingsense. Various modifications of the disclosed embodiments, as well asalternative embodiments of the invention, will become apparent topersons skilled in the art upon reference to the description of theinvention. It is therefore contemplated that such modifications can bemade without departing from the spirit or scope of the present inventionas defined in the appended claims.

1. A method of distinguishing domain-specific terms from a list specificto a particular domain comprising: extracting linguistic objects from aset of question-and-answer pairs wherein the linguistic objects includelexical answer types and answer entities, by executing firstinstructions in a computer system; grouping the lexical answer typesinto a first set and grouping the answer entities into a second set, byexecuting second instructions in a computer system; identifyingdistinguishing features of one or more corpora specific to theparticular domain wherein the distinguishing features distinguish thelexical answer types in the first set from the answer entities in thesecond set, by executing third instructions in the computer system; andclassifying the domain-specific terms as either lexical answer type oranswer entity based on the distinguishing features, by executing fourthinstructions in the computer system.
 2. (canceled)
 3. The method ofclaim 1 further comprising training a natural language classifier usingthe distinguishing features, and wherein said classifying uses thenatural language classifier.
 4. The method of claim 1 wherein thedistinguishing features include one or more syntactic features.
 5. Themethod of claim 1 wherein the distinguishing features include one ormore lexical features.
 6. The method of claim 1 wherein said extractinguses text analysis to automatically extract the sets of linguisticobjects.
 7. The method of claim 1 further comprising applying a lexiconof classified terms to a deep question answering system to yield ananswer to a question. 8.-20. (canceled)